Follow Slashdot blog updates by subscribing to our blog RSS feed

 



Forgot your password?
typodupeerror

Comment Re:Time to close the doors? (Score 1) 72

where are you going to get another 5000 people to track for a lifetime study of a once in a lifetime event?

Probably the same place the first study found its 5,000 people. The population is not shrinking that fast! However, why do you have to replicate the first study? Perhaps you can test a hypothesis from the first study using a smaller, targetted study or by specific analysis of previous studies? That's my point: exact replication does not teach us anything new, you learn a lot more from testing the claims of previous papers using better data or different approaches.

or where very specialized equipment that costs a small fortune to produce (like the stuff at CERN) are at play.

As a particle physicist who worked at CERN for many years you are completely wrong when you think that we do not reproduce and build on previous experiments. New accelerators generally have at least two major experiments on them, the LHC has ATLAS and CMS, which are completely independent of each other and have radically different detector designs in order for them to be able to confirm discoveries although the agreement is usually that when one experiment wishes to publish a major discovery the other experiment has a ~1 week period to indicate that they are ready to publish too: this is to prevent a mad rush to publish leading to shoddy work, simply to be first. For the Higgs AT:AS triggered the period and CMS, whose analysis was almost ready too, got ready fast and the discovery was jointly shared between the two experiments with two consistent and independent measurements.

However, the LHC did more than that because when you first turn on a new accelerator the first thing you do is establish the physics that we already know is there. This is what I meant by building on prior results. After the Tevatron found the top quark it would have been a waste of effort and money to build a second Tevatron to confirm that (although again there were two experiments DZero and CDF that both saw it). However, with even more energy the LHC can produce even more top quarks than the Tevatron and we both confirmed the Tevatron results and then improved on them. So we _do_ replicate previous studies, even in particle physics where accelerators are expensive, but we do not _just_ replicate: we also significantly improve on prior work.

Why do you insist that nothing is wrong, or that dedicated replication teams are so unglamorous

Exact replication is not "unglamorous" it is a wasted opportunity to improve on the result. If you are not going to add anything new to the sum of human knowledge then yes, your study is objectively much less valuable than a study which does improve our knowledge. It is also much easier to do. We should strive to do better than those who went before us, not to just do the same as they did!

the data suggests that fraud is INCREASING, and catching it is falling behind, which would indicate a failure in methodology...

Yes, fraud is increasing but it is connected to particular fields, generally medicine which is very importantly NOT science and where the differences make it much more susceptible to fraud because, while science's aim is understanding, medicine's aim is to cure people. This means that medicine is happy with correlation: give a person X and it cures them of Y is great medicine, whereas science's goal is causation: why/how does giving someone X cure them of Y? It's a subtle but critical difference between the fields which makes science much less susceptible to fraud.

Even ignoring the fact that, so far at least, the evidence that fraud is increasing is primarily confined to medicine rather than science, the fact that it is increasing is NOT evidence that science's methodology is flawed. It is simply evidence that there are more people willing to commit fraud out there and the fact that we are catching more is potentially evidence that our methodology is actually working - although not conclusive evidence because we do not know how many we miss e.g. because the results are irrelevant and so never used and hence tested by anyone.

Many of the cases, as the article you linked states, are associated with new open access journals that do not have proper peer review or academic standards in place, even for the ones that are not just out and out scams motivated by the shift to publisher-pays that lets scam journals earn money. Again, this is not due to scientific methodology failing but due to governments pushing an open access publishing model which, while it does have benefits, as we are finding out now also has some significant downsides. Specifically the high cost to publish in reputable open access journals is driving some researchers in poorer countries to turn to these cheaper, predatory journals that have little to no peer review.

None of these are problems that replication studies will in any way shape or form help with.

Comment Ambiguity (Score 1) 124

On the flip side even a fragmented sentence can perform a variety of complex tasks which would replace multiple clicks.

Theortically yes, in practice no because the other difference between a mouse click and a human sentence is that a mouse click is specific: if I right click a file and select delete it is unambiguously clear that I mean to delete that file. If I tell the computer "Delete the file." that sentence does not specific which, exact file I want to delete. To do that I have to specify an exact path or I have to first open a folder which again requires specifying which exact filder you want to open.

Then there are the ambiguities of spoken language e.g. how do I verbally differentiate between "J.py" and "Jay.py" or the "C", "See" and "Sea" directories? Human language is simply not designed with the precision that a computer requires and adding that specificity lowers the bandwidth considerably.

Comment Re:Input Bandwidth (Score 1) 124

Input bandwidth is not determined by the speed of the CPU but by the speed of the medium: connect a modern CPU to an old 56k modem and the bandwidth is still 56k. While you might be able to increase the effective bandwidth with compression you are limited to how much you can do that when the input source is human vocalization since the human needs to run the same compression algorithm. I doubt you'll do better than word-macros and, as I pointed out I can click a button to launch the same macro much faster than I can say a word.

Comment Re:Time to close the doors? (Score 1) 72

First, impact factor (not impact score) is used to compare journals, not individuals. Second, it is the average citations of every paper in a journal so one paper that gets a high citation rate through fraud will have minimal effect on the impact factor...and if you have so many fraudulent papers that it does then you are going to have a widepsread reputation in the field as as the journal if science-fiction at which point it will not matter what your imapct factor is, you'll be judged on that reputation.

The politcians I mention provide an insufficient financial resource to provide for the degree of replication needed, replication scientists dont get near the impact scores of seminal paper authors

Which is exactly how it should be because if all you have done is exactly reproduce someone else's work nobody has learnt anything new so the impact is close to zero. This is the sort of project you give to an undergrad, not something you would want to publish. Simply reproducing a previous work is a wasted opportunity to improve on it. Instead of aiming for replication, aim for improvement: increase the precision of the measurement or use an entirely different method that tests different assumptions. Replication usually happens naturally when people start to build on your result and improve it. This is why, despite as you point out there being no funding specifically for replication, we still catch scientific fraud.

Comment Re:Just like drugs (Score 1) 50

that the one person who has trained an AI in this discussion... is the person who you think doesn't know AI.

You are NOT the one person who has trained AI in this discussion, I have been using it for the last 20+ years. I'm not afraid of AI at all but, unlike you, I am very much aware of its limitations because, also I suspect unlike you, I have actually trained and used AI systems. However, given that you failed to read that in my post I suspect you may be the one person in the discussion who can't read and understand English which is another reason to doubt your claims.

Comment Peer Review (Score 2) 72

That's called peer review, and Mark Smyth passed all peer review. Yet he still pumped out fake papers; hundreds of them, polluting scientific knowledge with fake data.

Peer review is not the same as a fraud investigation. When we review papers we start from the assumption that the data in the paper was collected "honestly" i.e. that the researcher accurately reported to the best of their ability what they did and the data they collected. We then look at that data to ensure it looks consistent with what they did and that the method did not contain anything that might cause misleading data. Then we check that the conclusions in the paper are consistent with the data and analysis.

Peer review is there to prevent a researcher from fooling themselves (and others) by making claims that are not warranted by data or missing some subtle effect that could explain what is going on without the need for new science etc. It cannot check that the data are real although sometimes it can catch fraudulent data if that data are not consistent with well-known and established science. However, fakers are usually smart enough to be able to mske the data look consistent enough so it can be very hard to spot that in peer review. Ultimately it will get spotted as others try to reproduce results, faill and then start to look in much, much more detail at previous claimed results but peer review can't spot things at that level of detail.

Comment Not Needed: Good Journals Known (Score 2) 72

Journals are already "ranked" according to their "impact factor", which is a number calculated based on how often their articles are cited by other articles; it would make sense to also calculate a "credibility factor"

Impact factor generally is a credibility factor or at least I do not know of any journal in my field where there is a low-credibility journal with a high impact factor, although there are some specialist journals - e.g. instrumentation - which are highly credible but with a low impact factor. Generally speaking though anyone in the field worth their salt will know which the good journals are and where a paper is published generally does have a large impact on how we regard its quality.

I do not see a good way for a "credibility factor" to be calculated in an objective manner that would not have significant negative repurcussions e.g. counting the number of retractions would be bad since it would encourage journals never to retract papers. Similarly even the best intitutes can hire rogue researchers - or more commonly have bad grad students or postdocs - and enouraging journals to accept anything from any researcher in a "respected" instistute to boost their credibility would be bad too. Also papers in many fields cannot and do not have a single "primary" author.

Comment Re:Time to close the doors? (Score 2) 72

Currently, the paradigm is 'publish or perish', because science funding is only handed out to 'rockstars' by politicians

That is utterly wrong. As a scientist who has sat on several grant review boards there are no politicians involved at all in deciding who gets funding. The politicians set the size of the pot we have to give out but grant applications undergo rigorous, multi-stage peer evaluation. Even in the US where a single expert program officer has a lot of control over a grant program (or at least they used to) peer evaluation was still critical to the process. The only exception to this are "mega-projects" where the cost is so significant that it merits a line-item in the national budget and then yes, politicians obviously have to be involved but this is not where the vast majority of research funding comes from and at that point they are listening to the views of multiple experts and weighing in the national and political interests, not counting papers.

When grants are peer reviewed nobody just looks at the number of papers if the appilcants and goes "oh wow that guy published X papers lets give him everything he asked for!". Instead we look at the quality and impact of that work as well as what they are actually proposing. Different people weight these things differently - I tend to weigh the proposal more, others weigh past pulication record higher and both are very valid. However, in evaluating publications we use things like venue of publication (how many are in top journals for the field?) and citations (h-index) - although even then you have be to careful since that depends a lot on the field. Rate is a consideration but large numbers of papers in dodgy journals will count for nothing, indeed they would be detrimental since those reviewing it would be asking what he person is up to and how can they not know that the journals they are publishing in are trash.

Comment Input Bandwidth (Score 4, Insightful) 124

It will not be the chatter that kills this but the input bandwidth. Even if you assume it would allow you to set up some "verbal macros" to execute when a single word is spoken I can still click mouse buttons faster than I can speak words. The same goes for output bandwidth but even more so - it is much, much faster to see diagrams, buttons and read text then it is to listen to the computer speak information.

I can see this being useful in limited applications - such as in-car systems where a verbal inface and lowing bandiwicth would be a huge benefit. However, I cannot see it replacing a regular desktop/laptop OS.

Comment Just like drugs (Score 4, Insightful) 50

I've built one from scratch. ...Telling me I don't know AI is well... funny.

I've trained many machine learning models as well, from BDTs through to GNNs but never an LLM - although arguably not entirely from scratch (except for an early BDT) since we used existing libraries to implement a lot of ML functionality and once setup we just provided the training data. If you really have trained an LLM "from scratch" as you claim then surely you must be aware of how inaccurate they can be? I mean even the "professional grade" ones like Gemini and ChatGPT get things wrong, omit details and make utterly illogical inferences from time to time.

I'd agree with the OP that you do not know AI - even if you are capable of building an ML model from scratch (I presume using a suitable toolkit so not realy from scratch) you clearly do not understand the reliabilty of its output or are incapable of seeing how that might be a serious problem when advising someone with mental health issues which raises questions about exactly how much you understand of what you might be doing.

The new law seems to be well written. All it does is ensure that a medical professional has approved the use of the system. It's the same type of protections we have for drugs, we do not just let companies release any drug they like before it has undergone testing and a panel of medical experts agrees that it is both safe and effective and even then they do not always get it right! How is it stupid to have similar protections for computer software used to treat mental health problems? It does not prevent you from using software in this way all it requires is that an expert has said that it is safe and effective.

Comment Re:So maybe... (Score 2) 87

I'm not a fan of AI getting used in marketing/advertising at all. But that's mostly because I find most of it can still be picked out from reality.

That's probably why it is so useful for advertizing: the goal there is not to reproduce reality but an idealized approximation of it and some of the photoshopping ad companies have done in the past has produced results far worse than current AI is capable of.

Comment Monarchs Nerfed before US Revolution (Score 1) 163

England was never going to nerf its monarchy if we were still saying "long live the king!" from across the pond.

Actually we "nerfed" the monarchy in 1649 while you were still part of the UK and still saying "god save the king!" from across the pond. It happened as a result of the English civil war that established parliament's pre-eminence over the monarchy - and the "nerfing" was pretty severe since Charles I was beheaded! While the monarchy was restored in 1660 it was as a figurehead position with little to no political power, or as you would put it, a severely "nerfed" version of what went before!

Even if you had not rebelled though you would almost certainly not be ruled over by the UK government by now, in the same way that the UK government has absolutely no control over Canada. Canada is a completely separate nation from the UK that just happens to have the same monarch as the UK. The two titles: King of the United Kingdom of Great Britain and Northern Ireland and King of Canada are entirely seperate and equal. However, I doubt Trump would be interested in a position as king though, while they do get a degree of deference, UK monarchs have not been able to rule by royal decree since we "nerfed" them and they are subject to the law.

Comment Vulnerable or Enhanced? (Score 1) 163

Mathematician is a valid job just like any science specialization. What I fail to see is how these jobs are really vulnerable to AI. I can definitely see that the job will be transformed by AI - much like physics has been transformed by using machine learning for data analysis - but I do not see much chance that it will be eliminated by AI.

What I see happening is mathematicians using the powerful capabilities of AI to do more just like almost every other branch of science has so I see it more as tool that enhances the capabilities of people in such jobs, not as something that will replace them.

Slashdot Top Deals

How can you do 'New Math' problems with an 'Old Math' mind? -- Charles Schulz

Working...